Skip to content

Add moments collection + Mnemosyne sync workflow#4

Open
trustbutverafy wants to merge 1 commit into
eigenlucy:masterfrom
trustbutverafy:moments-collection-and-mnemosyne-sync
Open

Add moments collection + Mnemosyne sync workflow#4
trustbutverafy wants to merge 1 commit into
eigenlucy:masterfrom
trustbutverafy:moments-collection-and-mnemosyne-sync

Conversation

@trustbutverafy

Copy link
Copy Markdown

What this is

A starting point for syncing short narrative fragments from Vera's
Mnemosyne platform into a new Jekyll _moments collection on your site.

You write fragments in Mnemosyne (in a folder named moments), and a
daily GitHub Actions job exports them as .md files into _moments/,
commits them, and rebuilds the site. Hand-written .md files in
_moments/ are never touched by the sync — only Mnemosyne-synced files
are eligible for cleanup.

It's deliberately easy to turn off: delete the schedule: line in
deploy.yml, or remove the sync-mnemosyne-folder job entirely, and
everything else (the collection, the layout, the index page) keeps
working as a normal hand-written Jekyll collection.


What's added

Jekyll collection

  • _config.yml — registers moments collection with permalink /moments/:path/
  • _layouts/moment.liquid — small layout (optional title, date, tags, content)
  • _pages/moments.md — reverse-chronological index page at /moments/, with nav_order: 6. Open question: do you want this earlier in the nav? See note below.
  • _moments/2026-05-13-first-moment.md — a hand-written sample fragment so you can see what the format looks like. No mnemosyne: provenance block on it, so the sync will never delete it.

CI sync

  • .github/scripts/sync_from_mnemosyne.py — Python script that calls Vera's platform, writes files, and reconciles deletes via provenance blocks. ~360 lines, only depends on pyyaml.
  • .github/workflows/deploy.yml — new sync-mnemosyne-folder job that runs before the existing deploy job, on either:
    • workflow_dispatch (manual button in the Actions tab), or
    • schedule: cron \"0 10 * * *\" (daily at 10:00 UTC)
      The synced commit auto-retriggers the deploy job via the existing push: path.

Ruby pin

  • .tool-versions — pins Ruby 3.2.2 to match CI. Works with mise (recommended), asdf, or rbenv.

How the sync works

┌────────────────────────────┐         ┌───────────────────────────┐
│  Mnemosyne graph: lucy     │         │  GitHub Actions runner    │
│                            │         │                           │
│  /                         │  POST   │  sync_from_mnemosyne.py:  │
│   ├─ moments/  ←───folder──┼─────────┼─→ fetch folder contents   │
│   │   ├─ \"fog on dolores\"  │         │   render as markdown      │
│   │   └─ \"late espresso\"   │         │   write to _moments/      │
│   └─ ...                   │         │   git rm any orphans      │
└────────────────────────────┘         │   commit + push           │
                                       └───────────────────────────┘
                                                  │
                                                  ▼  push triggers existing
                                       ┌───────────────────────────┐
                                       │  deploy job rebuilds      │
                                       │  Jekyll site as usual     │
                                       └───────────────────────────┘

Mnemosyne is the platform Vera runs (mnemosyne-platform repo). It returns a JSON batch:

```json
{
"folder": { "name": "moments", "id": "" },
"items": [{ "document_id": "...", "filename": "YYYY-MM-DD-slug-doc8.md", "content": "<frontmatter + body>" }],
"document_ids": ["", ...],
"warnings": [{ "document_id": "...", "reason": "empty after render" }]
}
```

The script writes each item to _moments/<filename>, then scans existing files and removes any whose mnemosyne.document_id isn't in the response's document_ids list. Files without a mnemosyne: block are always safe.


What you need to do to turn it on

  1. Get a Mnemosyne account with a paid tier (so you can mint API keys)
  2. Mint an API key — Mnemosyne Settings → API Keys → Create. Label it something like github-eigenlucy. Copy the secret immediately — you only see it once.
  3. Add GitHub repo secrets at https://github.com/eigenlucy/eigenlucy.github.io/settings/secrets/actions:
    • MNEMOSYNE_URL — base URL of the platform, no trailing slash (e.g. https://mnemosyne.dev.sophia-labs.com)
    • MNEMOSYNE_API_KEY — the mnemo_… secret from step 2
  4. Create a folder named moments in your Mnemosyne graph (top-level, label exactly moments)
  5. Write a fragment in that folder in the Mnemosyne editor
  6. Trigger the workflow — Actions tab → "deploy" → "Run workflow" with default inputs (graph_id=lucy, folder_name=moments, collection=moments)
    • Or just wait until tomorrow at 10:00 UTC for the scheduled run

If you'd rather start without the daily cron, delete the schedule: block in deploy.yml. The workflow_dispatch button still works for manual runs.


Safety properties

These are the load-bearing invariants — if any of these are wrong, please flag it:

  • Hand-written moments are sacred. The sync only deletes files that carry a mnemosyne: provenance block in their YAML frontmatter. Anything you write by hand stays put.
  • The sync never touches other collections. It targets exactly _<collection>/ (default _moments/). Your _posts/, _eigenlucy/, etc. are untouched.
  • The workflow only runs on the trigger you configure. No silent sync on every push. Pure workflow_dispatch + the daily cron.
  • No write-back to Mnemosyne. The endpoint is read-only on the Mnemosyne side; this is purely an export.
  • Failed renders don't break the workflow. Empty docs are reported as warnings; the rest of the batch still publishes.

What's been tested locally

  • ✅ End-to-end round-trip on a local Kubernetes cluster: TipTap doc in Mnemosyne → endpoint → script → file written → orphan deleted → Jekyll built → rendered at /moments/<slug>/
  • ✅ Hand-written file with no mnemosyne: block survived a sync where its filename wasn't in the response
  • ✅ Hand-written file with a faked mnemosyne: block got correctly flagged as orphan and removed
  • ✅ Empty Mnemosyne docs surfaced as warnings, not failures
  • ✅ Jekyll build green with the new collection (no warnings introduced by our additions)

Known quirks (not blocking, just FYI)

  • Body content sometimes duplicates. Mnemosyne's existing tiptap_xml_to_markdown converter has a pre-existing bug where the same paragraph can appear twice in exports. This affects both the new /generate-markdown endpoint and the existing /documents/.../export endpoint identically. Vera is filing it separately. Probably only triggers for docs created in unusual ways; docs you write in the Mnemosyne editor likely look fine. If you see duplication, that's why.
  • Filenames look ugly without a heading. The slug part of the filename comes from the doc's first heading block. If your moment starts with a plain paragraph (no # heading), the filename falls back to a chunk of the document id — e.g. 2026-05-13-5ada439e-5ada439e.md. Cosmetic; the page renders fine. Start moments with a heading if you want pretty filenames.
  • No deletion-from-Mnemosyne path yet. If you delete a doc from Mnemosyne entirely (not just move it out of the moments folder), the next sync still deletes its file on the Jekyll side via the orphan logic. That works. But there's no audit trail — the file just disappears. Worth knowing.
  • nav_order: 6 for the moments index puts it after eigenlucy in your nav. If you want it earlier (say, between blog and resume), bump the other pages — happy to do that in a follow-up.

Companion change

The endpoint lives in mnemosyne-platform on branch feature/generate-markdown-jekyll-export. That branch needs to be deployed before this workflow can do anything — until then, the workflow will fail with a connection error, which is fine (it won't break your site).


How to revert if needed

The whole sync is contained. To turn it off:

  1. Stop scheduled runs: delete the schedule: block from .github/workflows/deploy.yml (or just the cron line). The deploy job keeps working.
  2. Remove the integration entirely: revert this PR. The _moments/ collection is just a Jekyll collection at that point — hand-write files in it like any other collection, or delete it.
  3. Temporarily disable: rename the workflow file or comment out the sync-mnemosyne-folder job.

You don't lose any content — synced files have already been committed to git history, and hand-written files were never touched.


🤖 Generated with Claude Code

Introduces a new `_moments` Jekyll collection for short narrative
fragments, with a CI workflow that syncs documents from a Mnemosyne
folder into the collection on a daily schedule (or via manual dispatch).

- `_moments/` collection with dedicated `moment` layout and index page
- `.github/scripts/sync_from_mnemosyne.py` calls the platform's
  POST /generate-markdown/folder endpoint, writes synced files, and
  removes orphaned ones via provenance-block reconciliation
- `.github/workflows/deploy.yml` adds a `sync-mnemosyne-folder` job
  (workflow_dispatch + daily cron at 10:00 UTC) that runs before deploy
- `.tool-versions` pins Ruby 3.2.2 to match CI

Hand-written moments are never touched by the sync — only files with a
`mnemosyne:` frontmatter block are eligible for reconciliation.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant